Hierarchical Attention Network For Multilabel Classification (Detailed Case study)

Multilabel Text Classification Using Custom Embeddings in Keras.¶

Traditionally, classification problems are either formulated as -

1. Binary Classification problems - where each training example belongs to either a positive or a negative class.
2. Multiclass Classification problem - where each training example belongs to one of the 'k' classes.

In this article, we shall explore a new problem statement -

Multilabel Classification Problem -¶

• In many real world data applications, often we encounter scenarios where each data point belongs to multiple classes.
• Suppose you are working for Recruiting Firm which receives thousands of Resumes each day. You are tasked with the problem of tagging each resume with relevant set of skills which hiring committee cares about.
• In such a scenario, it is possible to have a candidate who possesses 'leadership' skills as well as 'coding' skills.
• Such a setup requires you to solve a multilabel classification problem.

The following sections will cover -

1. Data Description,
2. Data Cleaning.
3. Encoding Labels.
4. Defining a custom Embedding Layer with keras.
5. Defining Loss Function and Evaluation Metric for Multilabel Classification
6. Model Building
7. Training a model
8. Inference.
9. Final Comments and References.

Let's start by importing a whole bunch of python libraries that we will need for successful implementation of our algorithm.

Data Description -¶

For implementing a multilabel classifier, we shall be using the "DBPedia Hierarchical Classes dataset" dataset from Kaggle.

Context -¶

• DBpedia (from "DB" for "database") is a project aiming to extract structured content from the information created in Wikipedia.
• This is an extract of the data that provides taxonomic, hierarchical categories ("classes") for 342,782 wikipedia articles.
• There are 3 levels, with 9, 70 and 219 classes respectively. A version of this dataset is a popular baseline for NLP/text classification tasks.

• This version of the dataset is much tougher, especially if the L2/L3 levels are used as the targets.

• The Dataset consists of the following columns -

  1. text - a free text column which describes the entity.
  2. l1 - Base label in the hierarchy.
  3. l2 - Intermediate label in the hierarchy.
  4. l3 - Top level label in the hierarchy.

There can be multiple tags associated with each entity making it a multilabel classificatiopn problem.

Data Cleaning -¶

In this section, we shall perform some basic preprocessing on our raw text data using following techniques -

1. merge title and synopsis data columns for better handling. We shall refer this column as raw_text
2. Split comma separated tags for each movie.
3. Remove punctuation and stopwords from raw_text.
4. Lemmatize raw_text - Lemmatization usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . For more information on lemmatization refer to the stanford notes.

Train Test Split -¶

In this section, we shall split our data into 3 subsets -

a. Training data - This data will be used for training our classifier my minimizing the overall loss.
b. Validation data - The Validation data will help in selecting hyper-parameters. Since this data is not explicitly used for training the model, validation data also helps in reducing overfitting.
c. Test Data - We shall use test data to estimate model's performance on OOB (out-of-bag) data points.

Since there is no off the shelf function in sklearn or keras that splits dataset into 3 different subsets, We shall leverage the train_test_split function by making a call to it twice.

Label Encoding -¶

1. In order to train a classifier, we need to convert the string tags into a vector of 1s and 0s.
2. There are about 71 tags in the dataset.
3. Therefore, for each datapoint, we need to construct a 1 x 71 dimensional vector with each bit in the vector representing a tag.
4. Since datapoint can have multiple tags, the target label will consist of 1s at all indices corresponding to the target tag attached to the datapoint.

To accomplish this, we shall train a MultiLabelBinarizer.

Custom Embedding Layer -¶

1. Embedding layer in keras is responsible of representing the input sequence of text with a vector of a size defined in the model.
2. one of the most adopted techniques while working with text is to train a tfidf vectorizer followed by a Dimensionality Reduction technique (PCA, Random Projections, etc) .
3. The embedding layer in keras facilitates the execution of sequence mentioned above while training a classifier.
4. It is also possible to derive weights from a pretrained model like word2vec or gloVe. However, in this article we shall focus on training a custom Embedding layer.
5. The Embedding layer expect a fixed input size of dimensions (batch_size $\times$ max_sequence_length) where max_sequence_length determines the maximum length of input text sequence.
6. The Output of Embedding Layer is a matrix of shape batch_size X max_sequence_length X embedding_size.

In order to match the input contract, we shall train a keras Tokenizer using fit_on_texts function. Note that, one can use different techniques to identify the max sequence length. For the sake of this project, we will consider the 90th percentile of the input sequence length distribution.

Attention Mechanism -¶

Before we proceed further, I would like to take a pause and delve into the philosophical and mathematical details of Attention Mechanism.

Why Attention Mechanism ?¶

• Consider a simple spam vs not-spam binary text classification problem where input is a free flowing email text and you would like to predict if it is a spam or not-spam.
• Typically, if you were to decide manually, you invariably look for trigger words (Nigerian Prince, Lottery Winner, No credit check, <suspicious link> , etc) which would hint towards an email being spam.
• These trigger words influence your decision no matter where they appear in text. In other words, subconsciously you end up assigning a higher weight to these trigger keywords which ultimately helps you reach a decision.
• Attention Mechanism tries to replicate a similar behaviour where it attempts to assign a higher weightage to trigger keywords that ultimately helps our model in reaching a decision.

Aren't the likes of GRUs and LSTMs suppose to do something similar?¶

• Well, a typical RNN network suffers from the problem of vanishing gradients. As a result, earlier parts of a temporal data is unable to contribute towards the final decision being made.
• GRUs and LSTMs are able to ameliorate this problem by introducing memory cells and forget gates.
• However, they can remember sequences of 100s but not definitely not 1000s.
• Sure, one can take a Global Max of returned activations and use them to predict classes but a "hard-max" takes the maximum value and forgets everything else.

ENTER ATTENTION MECHANISM !!

• A softmax (Attention weights) gives the probability distribution over each activation at different time steps. Definitely, a distribution is more helpful than a point estimate (Stats 101, right?)

Estimating Attention Weights while training -¶

• Hopefully, the need for an Attention Mechanism is justified in the previous section.
• In this section, let's go over some of the mathematical details to better understand the concept. This will help us while building the network as well.

Let's get to it -¶

• It is evident that these weights are dynamic and hence cannot be pre-computed.
• The following image by WildML summarises the attention mechanism in a block diagram format.

• As seen in the above image, the output at time 't' is a weighted average of weights $\alpha^{<t,t'>}$ where t' = ${1, 2, 3....T_x}$
• $\alpha^{<t,t'>}$ indicates the amount of attention that $y^{<t>}$ should pay to $a^{<t'>}$
  
• \begin{align} c^{} = \sum_{t'=1}^{Tx} \alpha^{} * a^{} \end{align}

How to compute $\alpha^{<t,t'>}$ ?¶

• Since $\alpha^{<t,t'>}$ are weights at time t representing the probability distribution of attentions, a good way to model them would be to use Softmax (If you guessed it, be proud of yourselves !!)

Let's define -

• \begin{align} \alpha^{} = \frac{\exp{e^{}}}{\sum_{t=1}^{T_x} \exp{e^{}}} \end{align}

• It is easy to conclude that where model should pay attention is dictated by the values of $<e^{t,t'}>$

• Where to pay attention depends not only on hidden states ($a^{<t'>}$) but also on where we are in the output ($s^{<t-1>}$) sequence of length $T_y$.

• However, we don't know the function that maps hidden states and current output to $e^{<t,t'>}$. Hence we shall determine the function using a small neural network.

• In text classification, the output steps are limited to 1. This simplifies our problem statement. The magnified block diagram for attention mechanism is as follows -
  

Hierarchical Attention Network -¶

• With the basics around attention network chalked out, let's take this a notch higher.
• Hierarchical Attention Networks were introduced by Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, & Eduard Hovy at Carnegie Melon in association with Microsoft.
• The proposed model exploits the herarchical structure in documents.
• Each document can be viewed as a group sentences with each sentence as collection of words.
• The block diagram of the architecture is as follows -

• The model represents a document representation by building a representation of sentences which are then aggregated.
• Further, the model also represents each sentence by building a representation of words.
• This takes into account the fact that words as well as sentences are differentially important based on context.

The original paper can be found here. Do take some time to read it.

Data Wrangling -¶

• It is now time to implement the ideas studied in earlier sections.
• We start by pre-processing our input data.
• The raw input data consists of paragraphs of textual information of dimensions m $\times$ 1.
• In the next section, we simply convert of 1D data into 2D. The new dimensions will be $m \times n_{sent} \times n_{words}$ where m is the number of training examples, $n_{sent}$ is the maximum number of sentences in each training example and $n_{words}$ is the maximum number of words in a sentence.

Transfer Learning -¶

• To expedite the training process, we shall piggy back on pre-trained gloVe models.
• The process of building Embedding matrix is pretty standard.

Loss Function and Evaluation metric -¶

Loss Function -¶

• A cross entropy loss for C classes and m training examples is defined as -

\begin{align} CE = - \sum_{i = 1}^{m} \sum_{j= 1}^{C} y_ilog(\hat{y_i})\end{align}

• Typically a Binary classification problem uses a sigmoid activation in the final layer of the neural network.
• The sigmoid function for pre-activations $z$ is defined as -

\begin{align} \hat{y} = \frac{1}{1 + e^{-z}} \end{align}

     Where,  

\begin{align} z = WX + b \end{align}\begin{align} X = m \times n \end{align}\begin{align} W = n \times 1 \end{align}

The cross entropy loss function for binary classification can hence be written as - \begin{align} BCE & = - \sum_{i = 1}^{m} \sum_{j= 1}^{2} y_ilog(\hat{y_i}) \\ & = - \sum_{i = 1}^{m} y_i log(\hat{y_i}) + (1- y_i)log(1 - \hat{y_i}) \end{align}

• For multilabel classification we shall assume that each label as an independent Binomial Random Variable.
• As a result of this assumption, a loss of Binary loss function makes much more sense than a categorical cross entropy function.
• Lastly, by the assumption of independence, since the probability that a data point belongs to class i is independent of whether of class j, the activation in the final layer of the network has to be "sigmoid" and not "softmax".

For a multilabel classification problem, various different loss function can also be used. Check them out at the following links for more information.

1. Multilabel Classification - Label Dependence, Loss Minimization and Reduction Algorithms by Krzysztof Dembczy´nski
2. Distance for Binary Variables
3. Approaches for the Improvement of the Multilabel Multiclass Classification with a huge Number of Classes by Martha Tatusch

Evaluation Metric -¶

• Typically, a standard metric for classification like accuracy can suffice.
• However, it can often be misleading when target labels have only a small subset which is positive. In such cases, accuracy is often over-estimated.
• To overcome this problem, we shall be using a custom loss function defined by Matthew R. Boutell in Learning multi-label scene classification
• Please find below a snippet from the paper which describes the $\alpha evaluation$ metric -

Building HAN Model -¶

• In order to build a Hierarchical Attention network, we will split it into two models for better visibility, debugging and code management.
• First, we define a "word attention model" which takes a sentence as input and return activations based an attention mechanism.
• Next we design a "sentence attention model" which is responsible for representing attention mechanism for each sentence at the input.
• Note that, we shall be using TimeDistributed layer provided by keras to extract word attention representation for each sentence at the input. The TimeDistributed layer ensures that same weights are applied to each input in the sequence.

Training Analysis -¶

• It can be seen that the loss is decreasing consistently across epochs.
• The custom_score keeps getting better.
• The custom_score and loss on validation dataset is comparable with training phase which indicates that the model is not overfitting.

Investigating Attention Mechanism -¶

• In order to gain more insights on how our model is training, let us perform a quick analysis of how model is estimated weights and which words are actually being paid attention to while predicting multilabel classes.
• To facilitate this analysis, we have tapped our original models to spit out attention weighted activations.
• Based on these activations, we can back trace the word in the input which are influencing the final decision.

Without much further a do, lets jump into the implementation !!

Results and Final Comments -¶

VOILA !!! We can now trace the path of how the model is reaching its decision.

1. As seen from the output printed on the console, the input text is describing a wrestling event.
2. The model is iterating over different sentences and words with more "Attention" are being printed.
3. The model is able to correctly assign weights in each word which is indicative of wrestling despite having other words like philadelphia, pennsylvania, and unites states which hints towards "Place" as a potential label.
4. By assigning a lower weights to words that are not so important in the overall context of the document, the model is more robust and accurate.

Final Comments -¶

1. To scope down the problem statement, the output sequence in our problem definition has a size 1.
2. Hence it suffices to ignore the previous decoder states while estimating attentions at time t.
3. In other words, $T_y$ in our case is 1 which largely simplifies the implementation.
4. The code will change if we are tasked with a use case that involves a temporal output. In such a situation, we will have to account for previous decoder states while estimating attentions.

5. Finally, this article covers a lot of new concepts viz. -

   a. Formulation of Multilabel classification
   b. Attention Mechanism
   c. Hierarchical Attention Network
   d. Loss function and Activations for multilabel classification problem
   e. custom loss function.
   

Some of these may not be easy to digest. Feel free to reach out to me if you have any queries or would like to brainstorm ideas.